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The charge to this author was to address 3 questions: 0,^0,.^ 
1. What is evaluation utilization? 



2. How do you determine utilization? to the educational resources 

INFORMATION CENTER (ERIC)." 

3. What are implications for practice and research? 

Answering the first question requires definition of what is meant by 
"utilization." For the second question an analysis of research methodologies 
is needed. Given analyses and answers to the first two questions, it seems 
reasonable that through extrapolation implications can be drawn for evaluation 
practice and research. 

Before proposing answers to these three questions it is important to 
point out that I do not see myself as an "evaluator." Rather, my research 
examines the change process in schools and colleges. In this research we are 
attempting to develop understandings about how individuals experience change 
within organizational contexts. Thus, I bring a different set of assumptions 
and perspectives to the question, what is evaluation utilization? At the same 
time it appears to me that the concepts and tools that have been developed out 
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of Change process research have direct application in evaluation. Thus, the 
title, viewing evaluation utilization as an innovation. 

The Change Process and Utilization 
Since n\y training and work are not in evaluation a first step in 
preparing this paper was to review some of the recent evaluation literature. 
In addition, I talked to a non randomly selected sample of evaluators. This 
was done in an attempt to develop a better understanding of the utilization 
issues that evaluators are presently addressing. Another reason for the 
survey was to develop an understanding of how evaluation utilization has been 
defined by evaluators. 

Evaluator's Definition of Utilization 

Two useful references that were suggested to me were. Utilizing 
Evaluation , edited by James A. Ciarlo and Using Evaluations; Does Evaluation 
Make a Difference by Alkin, Daillas and White. The chapter by Carol Weiss in 
Ciarlo was particularly helpful. 

Until recently evaluators seem to have defined utilization in two ways. 

a) Utilization has been defined in terms of an analysis of the decisions 
that were made by managers. The more closely decisions follow the trends and 
implications of the evaluation data the greater the utilization. 

b) The second way of determining utilization is through examination of 
, the "impact" of the decisions. Within this way there appears to be some 

debate about whether one looks only at immediate and/or long-term impacts of 
the decisions. 

It apj)ears that utilization has been defined in terms of consequences of 
jjse of the evaluation data. That is, decisions are analyzed or the impact of 

decisions are analyzed, to see whether or not they followed the suggestions 
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made in the evaluation. Utilization is defined in terms of decisions based on 
evaluations and observable consequences of these decisions. There also seems 
to be some value implications that the bigger the consequences of the decision 
the more utilization there was. 

Evaluation as an Innovation 

For a change process researcher this emphasis upon analysis of decisions 
made a^d the impact of decisions was an interesting and somewhat curious way 
to define utilization. Since utilization, for someone who is studying change 
and implementation in organizations, focuses more upon use than it does the 
consequences of use. For change to be successful it is typically assumed that 
there will be use of the "innovation." 

The innovation is the process or product that is to be used. This may be 
something that is new or different or it may only represent a small adjustment 
to past practice. Typically an innovation includes descriptive materials, 
special training to give users the needed skills and there are persons who are 
seen as expert in use of the innovation. These experts provide training and 
consultation in how to use their innovation. Most experts are truly dedicated 
to their innovation and believe that most if not all prospective users should 
become users. 

From this frame of reference evaluations have the basic characteristics 
Of innovations . There are materials, processes, innovation experts and users 
need special skills to use them. An interesting change process study would be 
to research the adoption of evaluations as innovations. 

In addition, evaluators have many of the characteristics associated with 
developers of other innovations: 

They know their programs well and do quality work. 

They believe, truly believe, that evaluation is very important. 
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They have legions of followers who also truly believe. 

They have policy makers who will back them and use of their "innovation." 

They want their innovation used by everyone. 

And they have definite ideas about what ideal evaluation use means. 

Clearly use is problematic for evaluators a^.^ well as change process 
researchers. Evaluators are now experiencing what developers of many other 
innovations have been confronted with, how do you facilitate others in 
developing understanding of and use of evaluations? ' Spread of use of 
evaluation is now moving beyond the earliest more innovative adopters. , 
Evaluators are now being confronted with other adopter groups who are less 
interested in or less able to immediately use and accept the merit of 
evaluation (Rogers & Shoemaker, 1971). Evaluators are having to consider such 
change process variables as dissemination, adoption, implementation and 
institutionalization. 

In addition, as developers of other innovations have found out, very few 
adopters use evaluations in the ideal way. Adopters tend to make all sorts of 
adjustments (i.e. adaptations) and changes in the innovation, in this case 
evaluations. They may misinterpret or they may overly emphasize certain 
portions of the evaluation results and completely ignore others. Thus, 
evaluators as innovation developers are confronted with the problem of the 
degree of fidelity as well as integrity of use of their evaluation findings. 

In fact some users can really make a mess out of use of a particular 
innovation. There are times where it is quite conceivable that how evaluators 
and evaluation findings are used will be seen as "malpractice" by evaluators. 
With the increased spread in use of evaluations it is appropriate for 
evaluators to be concerned about utilization at this time. 



As the above discussion has attempted to illustrate, utilization is not a 
unique problem to evaluation. It is a problem that all innovation developers 
are confronted with. However, the utilization question still must be answered 
specifically for evaluation. Perhaps some of the research and concepts that 
have been developed in knowledge utilization and change research can be of 
help. 

Utilization as a Process 

With these ideas as context, lets examine how a change process researcher 
would define utilization. In doing this I shall also attempt to point out a 
couple of inconsistencies in how utilization has been defined in the 
evaluation literature. To do this, some description of the research that has 
been going on with the Concerns-Based Adoption Model (CBAM) (Hall, Wallace & 
Dossett, 1973) will be used. This research can help provide illustrations, as 
well as background for how evaluation utilization can be defined and why it 
might be useful to view evaluation as an innovation. 

Two of the basic assertions underlying the CBAM model are: 

(1) Change is a process , not an event. Implementation of an innovation 
does not happen instantaneously. There is a process that takes time and has 
its own natural pattern for unfolding. For examplo, the sending out of the 
final report or making an announcement at a board meeting does not result in 
instantaneous change. 

(2) The individual is the primary unit of adoption. Groups and 
organizations as a whole must be considered, but what individuals do will make 
or break most change efforts, Understanding how the individual views and uses 
the innovation is critical to facilitating change and assessing impact. 
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To translate change process research concepts to the domain of evaluation 
utilization is quite easy. Thus, the beginning definition of evaluation 
utilization would be: 

(1) Use of evaluation is a process , not an event. EvaViation findings, 
evaluation reports and recommendations cannot be delivered c;. a one time 
occurrence with the expectation that use will automatically follow. Rather, 
there is a process that can unfold where understanding, use and increases in 
sophistication about how to use evaluation gradually develop. 

(2) Use of evaluations must be assessed at the individual level as well 
as for the organization as a whole. I 

(3) Use must be considered across evaluations within an adopting 
organization, or across organizations for regional and national study. For 
most individuals and most organizations use entails more than understanding 
and making a decision or set of decisions based upon one report or the results 
of one evaluation study. Rather, use should be viewed across a series of 
evaluation reports and evaluation studies. 

One methodological implication of this view is that one time snapshots 
will not provide the needed information to judge utilization. Rather, the 
methodology for assessing utilization require cross-sectional and longitudinal 
designs. The sampling should be done across evaluation studies and reports 
within an organization or across a range of institutions if it is a study of 
large-scale use of one evaluahtion. 

Two Basic Questions 
In the Concerns-Based Adoption Model (CBAM), a great deal of attention is 
given to defining use. If evaluation utilization was the innovation to be 
studied using CBAM concepts and procedures then two basic questions would need 
to be asked of each_ftnd every potential adopter. 



(1) Is she/he using it? 

(2) What is it? 

To gather information about these two questions would entai^ assessing 
evaluation users/nonusers Levels of Use of the Innovation and their Innovation 
Configurations, 

Levels of Use 

To answer the first question, we would turn to the concept of Levels of 
Use of the Innovation (Hall, Loucks, Rutherford & Newlove, 1975). This 
concept grew out of our first studies of implementation at the individual 
level. Traditionally use had been considered to be a dichotomous variable, 
use or nonuse. The work with the Concern-Based model has led to 
distinguishing eight different Levels of Use (Lol!), three nonuse levels and 
five use levels. These Levels of Use are summarized in Figure 1. 

Basically the Levels of Use dimension describes the behaviors of 
individuals as they move from early exploration, to first use, to being 
relatively sophisticated in their use of the innovation. This is a generic 
concept and can be applied to any process or product innovation, including use 
of evaluation. 

Levels of Use is measured with a specially developed focused interview 
procedure (Loucks, Newlove & Hall, 1976). This interview appears to be a 
casual conversation with each interviewee. However, a trained Levels of Use 
interviewer can rate a person's Level of Use in relation to any oarticular 
innovation by using special questions and probes based on the Level of Use 
definitions and decision points listed in Figure 1. 

For this discussion one or two examples of these levels can be used to 
illustrate how they could be applied to the use of evaluation. For example a 
person at a Level of Use 0, Nonuse, is not doing anything relative to use of 



Figure 1 
Levels of Use of Evaluation 



Levels of Use Definition Behavior Example 



0 Nonuse 


State in which the user Evaluation report is not read, 
has little or no knowledge 
of the innovation, no in- 
volvement with the innova- 
tion, and is doing nothing 
toward becoming involved. 


Decision Point A 


Takes action to learn more detailed information about the innovation. 


I Orientation 


State in which the user has The evaluation report is read 
recently acquired or is ac- over. Some questions are asked 
quiring information about of evaluators^but the evaluation 
the innovation and/or has is not considered in decision 
recently explored or is ex- making, 
ploring its value orientation 
and its demands upon user and 
user system. 


Decision Point B 


Makes a decision to use the innovation by establishing a time to 
begin. 


II Preparation 


State in which the user is An administrator commits 
preparing for first use of resources to have an evalua- 
the innovation, tion done. 


Decision Point C 


Changes, if any, and use are dominated by user needs. 


III Mechanical Use 


State in which the user The principal calls a faculty 
focuses most effort on the meeting to consider the evalua- 
short-term, day-to-day use tion results, but he does not 
of the innovation with little fully understand them. Every- 
time for reflection. Changes one is confused and they reach 
in use are made more to meet no closure, 
user needs than client needs. 
The user is primarily engaged 
in a stepwise attempt to master 
the tasks required to use the 
innovation, often resulting in 
disjointed and superficial use. * 
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Figure 1- continued 



Decision Point D-1 


A routine pattern of use is established. 


IVA Routine 


Use of the innovation is The superintendent regularly 
stabilized. Few, if any, reviews evaluation information 
changes are being made in before presenting positions to 
ongoing use. Little prepa- the school board, 
ration or thought is being 
given to improving innova- 
tion use or its consequences. 


Decision Point D-2 


Changes use of the innovation based on formal or informal 
evaluation in order to increase client outcomes 


IVB Refinement 


State in which the user varies The director of staff development 
the use of the innovation to asks for a change in the workshop 
increase the impact on clients evaluation procedures so that she 
within the immediate sphere of will have information for indivi- 
influence. Variations are dual follow-up. 
based on knowledge of both 
short- and long-term consequences 
for clients. 


Decision Point E 


Initiates changes in use of innovation based on the input of and 
in coordination with what colleagues are doing. 


V Integration 


State in which the user is Several principals decide to 
combining own efforts to use pool their school evaluations 
the innovation with related and to work together with the 
activities of colleagues to evaluator to plan the use of 
achieve their own sphere of evaluation findings, 
influence. 


Decision Point F 


Begins exploring alternatives to or major modifications of the 
innovation presently in use. 


VI Renewal 


State in which the user An administrator returns from 
reevaluates the quality of a meeting where she has just 
use of the innovation, seeks heard about a new evaluation 
major modifications of or model that car- be used to test 
a iLerndL 1 ves to present inno- leacner competence ano sne 
vation to achieve increased wants to talk about its being 
impact on clients, examines used in place of the present 
new developments in the system, 
field, and explores new goals 
for self and the system. 



Based on the Levels of Use Concept. Hall, G. E., Loucks, S. F., Rutherford, W. L. 
O 4 Newlove, B. W. Levels ot use of the innovation: A framework for analyzing 
ERJC lnnojvation adoption. The Journal of Teacher Education , 1975, 26(1), 52-56. 
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evaluation. A report has been mailed to this person, but they do not read it. 
A meeting is called to describe the results of the evaluation study and this 
person does not attend. Their knowledge is minimal and no behaviors are being 
taken to learn more about or to use the results of the evaluation. 

This LoU 0 person can be contrasted with a person at Level of Use I, 
Orientation, who is exploring the evaluation findings and considering use. 
While a person at Level of Use II, Preparation has definitely decided to make 
use of the evaluation. 

We typically find that first time users will be at a Level III, 
Mechanical Use. This is a time where there is heavy dependence on the 
reference materials and a disjointed and inconsistent use of the innovation. 

Other persons might be observed to be at a Level of Use IVA, Routine Use, 
where use of evaluation has become a regular way and is stable over time. 
Other individuals may move to higher Levels of Use where various refinements 
and adaptations to increase outcomes are made. 

The Level i!> of Use concept focuses upon behaviors; what persons are 
actually doing with the innovation. In the case of evaluation utilization, 
users at these different Levels of Use would be behaving very differently with 
regard to the evaluation resources that were available to them. The person at 
Level of Use 0 would not be doing anything. The person at a Mechanical Level 
of Use might be making disjointed use of evaluation re^^orts and the evaluator. 
S/he might at one time make decisions that are consistent with the evaluation 
and another time be inconsistent. Later on their evaluation use may become 
more stabilized at a Routine Use. They have ways for regularly incorporating 
evaluatioa findings and evaluators into their work. Other evaluation users 
may start moving to higher Levels of Use and make refinements and adaptations 
in how they use evaluators. At higher Levels of Use they might ask the 
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Innovation Configuration Component Checklist 
for Individual Use of an Evaluation 



I. Review of Evaluation Reports 
(a) 



(b) 



Reports are read 
closely and find- 
ings can be quoted 



Some of the reports 
are emphasized and 
a few findings are 
quoted. 



(c) 

Reports are skimmed 
over. 



(d) 

Reports have been 
received but not 
read. 



(e) 

Knows nothing about 
the reports or their 
existence. 



2. 


Use of Evaluations 










(a) 

Evaluators are 
regularly asked 
for information 
and suggestior". 


(b) 

Evaluators are only 
involved when they 
deliver reports. 


(c) 

Rarely are evaluators 
talked with. 


(d) 

There is no com- 
munication with 
evaluators 


3. 


Role of Evaluation in 


Decision Making 








(a) 

Dominant influence. 


(b) 

One of multiple 
influences. 


(c) 

One of multiple, 
cumulative influences. 


(d) 

Not considered. 


4. 


Types of Decisions Where Evaluation is Used 








(a) 

Forming new policy. 


(b) 

Substantiating pre- 
vious decisions or 
actions. 


(c) 

Establishing or 
altering attitudes. 


(d) 

No clear pattern to 
use. 


5. 


Program Areas Where Evaluations Are Used (check all that apply) 





federal programs 
competency testing 
staff development 



teacher/administrator competency 

materials budget 

staffing 



plant operations 
other 
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evaluator to make changes in the design of future evaluations so that the 
evaluations could have even more impact on decisions and their activities. 

Innovation Configurations 

To answer the second question, "What is it?" we would turn to another 
CBAM concept. Innovation Configurations (Hall & Loucks, 1980). As our earlier 
studies focused upon implementation at the individual level, it became readily 
apparent that different users would adapt and sometime drastically mutate 
innovations as they used them. Sometimes the alterations were minor and 
sometimes they appeared to be major. In some instances it was not clear that 
any significant features of the innovation were in use, although the "user" 
might claim to be using the innovation. Thus, the concept of Innovation 
Configurations was created (Hall & Loucks, 1978). 

In this concept the operational form of the innovation is examined. Key 
components of the innovation are identified and how these components can vary 
are described. A trained observer and/or interviewer can gather information 
about how the innovation is being used by different persons and this 
information can be recorded in an Innovation Configuration checklist (Heck, 
Stiegelbauer, Hall & Loucks, 1981). 

It would seem that the concept of Innovation Configurations could be 
applied to evaluation utilization. Different individuals and different groups 
will use evaluations in different ways. Thus, if a list of components could 
be identified and descriptions of how each of these components varied were 
listed, a Configuration Checklist could be developed. This, checklist could be 
used to assess implementation of evaluation by different users. A very simple 
beginning for a configuration component checklist for evaluation is provided 
as Figure 2. The components In this checklist relay heavily upon parts of the 
definition of utilization proposed by Alkin, Daillak and White (1979). 
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Interviewers and observers would collect Information about how a 
particular evaluation was being used. The various components and component 
variations could be Identified and circled on the configuration checklist. 
The resultant pattern would represent one configuration of use of an 
evaluation study. Alternate configurations might be found for other users at 
the same site or other sites. In addition a composit component checklist 
could be developed to address organizational use. 

The next question Is, which configurations represent "ideal" use of an 
evaluation study, what others are "acceptable" and which configurations are - 
"unacceptable"? This question could be answered by the evaluators, It could 
be answered by the users or It could be made through some collaborative 
process. At any rate by having Identified various configurations of use of 
the evaluation It would be possible to describe operationally what use means. 

Once use has been determined In terms of the Level of Use and the 
Configuration that Is being used has been described, questions that have to do 
with effects of use can be explored. Such questions as what are the 
Influences of use upon evaluation decisions and what are the Impacts of these 
decisions can be explored. Some Intermediate questions could also be 
examined. How does the quality of decision making change as evaluation users 
change from being at Mechanical Use to Refinement use? Are certain 
configurations of evaluation use found more frequently at some Levels of Use? 
Are certain configuration components of evaluation use more easily adopted? 
The steps for the change process researcher In studying any of these questions 
would be to first assess use and describe what this means In operational 
terms. Then analyses of quality and consequences of use could be addressed. 
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Defining Utilization as an Innovation 
Three questions have been explored in this paper. This author's brief 
answers to these questions are: 

(1) What is utilization? 

Use is having each person a user (Level of Use III through VI) of certain 
configurations of evaluation use. Rather than viewing use as a dichotomous 
variable, use is defined in terms of what Level of Use each person is 
demonstrating and what component variations of evaluation use they are using. 
At all times the emphasis is on behavioral and operational descriptions of 
use. 

(2) How do you determine utilization? 

If use of evaluations is a process then the preferred study design is 
longitudinal, with cross-sectional designs being a second possibility. Use of 
evaluation must be assessed across time and evaluation studies within an 
organization. To judge use by observing use of one evaluation study at one 
point in time is equivalent to treating use as an event, rather than the 
process that it is. For across organization considerations, perhaps it is 
possible to look at use pf one evaluation, but there would need to be some 
form of stratification to be sure that the sample has a range in experience 
and sophistication of users. 

To assess use would entail conducting focused interviews to assess Levels 
of Use and development of an Innovation Configuration Checklist that could be 
assessed through a combination of interview and observation procedures. The 
resultant data could be analyzed to identify and describe the various 
configurations of the evaluation that were being used. 
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(3) What are some implications for practice and research? 

There are several implications that develop out of the above, a few of 
the more salient ones are briefly described next. 

(a) Use has to be documented first hand. Drawing inferences about how 
evaluations are being used by assessing a select few key decision makers or 
even more remote policy makers will not suffice. The first line users of the 
evaluation information will have to be pulsed to determine whether or not and 
how evaluations are being used. 

(b) Evaluators need to shift their perspective from seeing evaluation use 
as an event to its being a process. Users of evaluations do not 
instantaneously become expert, highly proficient and sophisticated in their 
use of evaluations. Rather, developing sophistication and understanding in 
using evaluations requires time and experience. Thus, evaluation use has 
similar characteristics to any other innovation where users gradually develop 
mastery. 

By viewing use of evaluation as a process, evaluators may find that with 
assistance and time inefficient or less able initial use of evaluations can 
lead to higher level use. The corollary also holds, more sophisticated users 
probably have had more experience in using evaluations. 

(c) Evaluators should recognize that evaluations and evaluators are 
innovations. When working with new clients all of the typical change process 
phenomenon are likely to occur. New adopters will be confused and will make 
mistakes. There will be some personal threat. The quality of decisions will 
not always be high. So it is important that evaluators develop their change 
process skills and be willing to invest the time it takes to get an 
"adoption". Delivery of the final report or a one time presentation of the 
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findings will not provide the necessary ongoing support to facilitate use of 
evaluations. 

(d) Early definitions of evaluation utilization appear to have confused 
the effects of use with use, as is illustrated in Figure 3. From a change 
process perspective evaluating utilization first entails determining whether 
or not there are users and what particular configurations of the evaluation 
are being used. Are evaluators consulted directly? Are evaluation reports 
read and understood? When decisions are made are there overt attempts to 
incorporate evaluation information into the decision making process? These 
are evaluation use questions that will provide descriptions of use. Then the 
C'ffects of utilization questions can be asked. 

Effects of use, entail examining the consequences of use of evaluations. 
A different set of variables need to be assessed here. Effects of use address 
such dimensions as the quality of decisions that are made, with and without 
evaluation, the number of actors that are involved in decision making, and the 
extent of change that results. 

It may be that some evaluators who have been considering the definition 
of utilization have implicitly defined utilization as effects only. The 
bottom part of Figure 3 is an illustration of how this would look. In this 
view use is subsumed under effects. Effects then become synonymous with use. 
Again, from the change process researcher's point of view it is important that 
evaluators distinguish between the use of evaluations and the effects of use. 
Evaluations could be incorporated into decision making in several different 
configurations and the effects of use of these different configurations could 
be quite different. 
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Figure 3 

Alternative Definitions of Utilization 



Use Leading to Effects 



USE 



USERS AT DIFFERENT 
LoU'S OF PARTICULAR 
CONFIGURATIONS 




EFFECTS 

TYPES AND QUALITIES 
OF DECISIONS 

"IMPACTS* 

SHORT- AND LONG- 
TERM 



Use Subsumed 

UTILIZATION 
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Hopefully these distinctions will help in clarifying some of the issues 
in analyzing and defining evaluation utilization.' In addition perhaps they 
will catalyze the establishment of closer linkages between the well developed 
literature in evaluation and the research that is being done on the change 
process. This may be a case where research from two different disciplines can 
be merged in a complimentary way to help both in understanding the phenomenon 
that they are studying. 
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